Use of Peripheral Component Interconnect Input/Output Virtualization Devices to Create Redundant Configurations

ABSTRACT

In one embodiment, a computer-implemented method for creating redundant system configurations is presented. The computer-implemented method creates a set of virtual function path authorization tables, and receives a request from a requester to provide requested data from a virtual function wherein the virtual function is performed by a single root or a multi-root peripheral component interconnect device. Further a receive buffer is created in a selected address range in a set of addresses ranges as well as a virtual function work queue entry for the virtual function containing an address of the receive buffer in the selected address range. Responsive to a determination that the virtual function is authorized, writing the requested data into the receive buffer of the selected address range in the one or more systems, and responsive to writing the requested data, issuing a notice of completion to the requester.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to an improved data processingsystem, and, more specifically, to a computer-implemented method, a dataprocessing system and a computer program product for creating redundantconfigurations using peripheral component interconnect input/outputvirtualization configurations.

2. Description of the Related Art

Typical computing devices make use of input/output (I/O) adapters andbuses that utilize a version or implementation of the PeripheralComponent Interconnect (PCI) standard, originally created by IntelCorporation in the 1990s, and now managed by the PCI-SIG. The PeripheralComponent Interconnect (PCI) standard specifies a computer bus forattaching peripheral devices to a computer motherboard. PCI Express, orPCIe, is an implementation of the PCI computer bus that uses existingPCI programming concepts, but bases the computer bus on a completelydifferent and much faster serial physical-layer communications protocol.The physical layer consists, not of a bi-directional bus which can beshared among a plurality of devices, but of single uni-directionallinks, which are connected to exactly two devices.

With reference to FIG. 1, an exemplary diagram illustrating a systemthat incorporates a peripheral component interconnect express (PCIe)bus, in accordance with the peripheral component interconnect expressspecification is presented. The particular system shown in FIG. 1 is ablade enclosure in which a plurality of server blades 101-104 areprovided. A server blade is a self-contained computer server designedfor high density systems. Server blades have many components removed forspace, power and other considerations while still having all thefunctionality components to be considered a computer. Blade enclosure100 provides services, such as power, cooling, networking, variousinterconnects, and management of various server blades 101-104 in bladeenclosure 100. Server blades 101-104 and blade enclosure 100 togetherform a blade system.

As shown in FIG. 1, peripheral component interconnect express isimplemented on each of server blades 101-104 and is used to connect toone of peripheral component interconnect express devices 105-112. Eachof these server blades 101-104 is then plugged into a slot in bladeenclosure 100 which then connects the outputs of the peripheralcomponent interconnect express Ethernet devices 105, 107, 109, and 111to Ethernet switch 113, via a backplane in blade enclosure 100, whichthen generates Ethernet connections 115 for external connectivity, forexample, communication connections to devices outside blade enclosure100. Similarly, each of the peripheral component interconnect expressstorage devices 106, 108, 110, and 112 are connected via the backplanein blade enclosure 100 to storage area network switch 114 which thengenerates storage area network connections 116 for externalconnectivity.

Thus, the system shown in FIG. 1 is exemplary of one type of dataprocessing system in which the peripheral component interconnect and/orperipheral component interconnect express specifications areimplemented. Other configurations of data processing systems are knownthat use the peripheral component interconnect and/or peripheralcomponent interconnect express specifications. These systems are variedin architecture and thus, a detailed treatment of each cannot be madeherein. For more information regarding peripheral component interconnectand peripheral component interconnect express, reference is made to theperipheral component interconnect and peripheral component interconnectexpress specifications available from the peripheral componentinterconnect special interest group (PCI-SIG) website at www.pcisig.com.

In addition to the peripheral component interconnect and peripheralcomponent interconnect express specifications, the peripheral componentinterconnect special interest group has also defined input/outputvirtualization (IOV) standards for defining how to design aninput/output adapter (IOA) which can be shared by several logicalpartitions (LPARs). A logical partition is a division of a computer'sprocessors, memory, and storage into multiple sets of resources so thateach set of resources can be operated independently with its ownoperating system instance and applications. The number of logicalpartitions that can be created depends on the system's processor modeland resources available. Typically, partitions are used for differentpurposes such as database operation, client/server operation, toseparate test and production environments, or the like. Each partitioncan communicate with the other partitions as if the other partition isin a separate machine.

In modern systems that support logical partitions, some resources may beshared amongst the logical partitions. As mentioned above, in theperipheral component interconnect and peripheral component interconnectexpress specification, one such resource that may be shared is theinput/output adapter using input/output virtualization mechanisms.

Further, the peripheral component interconnect special interest grouphas also defined input output virtualization (IOV) standards for sharinginput output adapters between multiple systems. This capability isreferred to as multi-root (MR) input output virtualization.

With reference to FIG. 2, an exemplary diagram illustrating a systemincorporating a peripheral component interconnect express multi-rootinput output virtualization is presented. In particular, FIG. 2illustrates how the architecture shown in FIG. 1 can be modified toshare the peripheral component interconnect express devices acrossmultiple systems.

Server blades 201-204 now generate peripheral component interconnectexpress root ports 205-212 and drive peripheral component interconnectexpress connections across blade enclosure 200 backplane, instead ofincorporating the peripheral component interconnect express devicesthemselves on sever blades 201-204 as was done with server blades101-104 in FIG. 1. The peripheral component interconnect express linksfrom each server blade 201-204 are then connected to one of multi-rootperipheral component interconnect express switches 213-214 which are inturn connected to peripheral component interconnect expressEthernet/storage devices 217-220. Peripheral component interconnectexpress Ethernet/storage devices 217-220 connect to the externalEthernet and storage devices through external connectivity 215 and 216.Thus, peripheral component interconnect express devices can be usedwithin blade enclosure 200. This reduces overall costs in that thenumber of peripheral component interconnect express devices 217-220 maybe minimized since they are shared across server blades 201-204.Moreover, this may reduce the complexity and cost of server blades201-204 themselves by not requiring integration of peripheral componentinterconnect express devices 217-220.

While the peripheral component interconnect special interest groupprovides a standard for defining how to design an input output adapterwhich can be shared by several logical partitions, the specificationdoes not define how to connect the input output adapters into a hostsystem. Moreover, the standard only specifies how each function can beassigned to a single system.

BRIEF SUMMARY OF THE INVENTION

According to one embodiment of the present invention, acomputer-implemented method for creating redundant system configurationsis presented. The computer-implemented method creates a set of virtualfunction path authorization tables, by a trusted entity, wherein entriesdefine access for a function to a set of address ranges in one or moresystems and the entries further defining a boundary preventing invalidcross function access, wherein the virtual function is performed by asingle root or a multi-root peripheral component interconnect device,receives a request from a requester to provide requested data from thevirtual function, creates a receive buffer in a selected address rangein the set of address ranges, and creates a virtual function work queueentry for the virtual function containing an address of the receivebuffer in the selected address range. Further, the computer-implementedmethod determines, in the set of virtual function path tables, whetherthe virtual function is authorized to use the selected address range,responsive to a determination that the virtual function is authorized,writes the requested data into the receive buffer of each address rangein the one or more systems, and responsive to writing the requesteddata, issuing a notice of completion to the requester.

In another embodiment, a data processing system for creating redundantsystem configurations is presented. The data processing system comprisesa bus, a memory connected to the bus, wherein the memory comprisescomputer-executable instructions, a central processor unit. The centralprocessor unit executes the computer-executable instructions to directthe data processing system to create a set of virtual function pathauthorization tables, by the trusted entity, wherein entries defineaccess for a function to a set of address ranges in one or more systemsand the entries further defining a boundary preventing invalid crossfunction access, wherein the virtual function is performed by a singleroot or a multi-root peripheral component interconnect device, receive arequest from a requester to provide requested data from the virtualfunction, create a receive buffer in a selected address range in the setof address ranges, create a virtual function work queue entry for thevirtual function containing an address of the receive buffer in theselected address range; determine, in the set of virtual function pathtables, whether the virtual function is authorized to use the selectedaddress range, responsive to a determination that the virtual functionis authorized, write the requested data into the receive buffer of theselected address range in the one or more systems, and responsive towriting the requested data, issue a notice of completion to therequester.

In another embodiment, a computer program product for creating redundantsystem configurations is presented. The computer program productcomprises a computer-readable medium having computer-executableinstructions stored thereon. The computer-executable instructionscomprise computer-executable instructions for creating a set of virtualfunction path authorization tables, by a trusted entity, wherein entriesdefine access for a virtual function to a set of address ranges in oneor more systems and the entries further defining a boundary preventinginvalid cross function access, wherein the virtual function is performedby a single root or a multi-root peripheral component interconnectdevice, computer-executable instructions for receiving a request from arequester to provide requested data from the virtual function, andcomputer-executable instructions for creating a receive buffer in aselected address range in the set of address ranges. Thecomputer-executable instructions further comprise computer-executableinstructions for creating a virtual function work queue entry for thevirtual function containing an address of the receive buffer in theselected address range, computer-executable instructions fordetermining, in the set of virtual function path tables, whether thevirtual function is authorized to use the selected address range,computer-executable instructions responsive to a determination that thevirtual function is authorized, for writing the requested data into thereceive buffer of the selected address range in the one or more systems,and computer-executable instructions responsive to writing the requesteddata, for issuing a notice of completion to the requester.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a block diagram of a system architecture implementing aperipheral component interconnect express standard;

FIG. 2 is a block diagram of the system of FIG. 1 incorporatingperipheral component interconnect multi-root input outputvirtualization;

FIG. 3 is a block diagram of a distributed computing system utilizing aperipheral component interconnect multi-root input output fabric;

FIG. 4 is a block diagram of the virtualization of system resourcesusing multiple logical partitions in which illustrative embodiments ofthe present invention may be implemented;

FIG. 5A is a block diagram of a peripheral component interconnectexpress multi-root input output virtualization enabled endpoint, inaccordance with an illustrative embodiment;

FIG. 5B is a block diagram of a peripheral component interconnectexpress multi-root enabled peripheral component interconnect expressswitch;

FIG. 6A is a block diagram of a virtual function work queue entry, inaccordance with an illustrative embodiment;

FIG. 6B is a block diagram of tables for validating the authority of avirtual function to access any given virtual hierarchy in a multi-rootdevice, in accordance with an illustrative embodiment;

FIG. 6C is a block diagram of a table for specifying an alternate routevirtual hierarchy for redundant path implementations of a multi-rootdevice, in accordance with an illustrative embodiment;

FIG. 6D is a block diagram of a table for specifying an authorizedaddress to virtual function relationship, in accordance with anillustrative embodiment;

FIG. 6E is a block diagram of a virtual function work queue entry usingan address of FIG. 6D, in accordance with an illustrative embodiment;

FIG. 7 is a block diagram of a configuration of redundant systems usingmulti-root devices and multi-root switches, in accordance with anillustrative embodiment;

FIG. 8 is a block diagram of a configuration of redundant logicalpartitions using a single root device, in accordance with anillustrative embodiment;

FIG. 9 is a flowchart of a process of multi-root fabric configuration ofan multi-root multi-system configuration in accordance with anillustrative embodiment;

FIG. 10 is a flowchart of a process for a system to determine thevirtual hierarchy numbers required for communicating to partner systems,in accordance with an illustrative embodiment;

FIG. 11 is a flowchart of a process to setup a virtual function workqueue entry in accordance with an illustrative embodiment;

FIG. 12 is a flowchart of a process for dynamically determininginput/output fabric path operational status and use of an alternate pathwhen necessary, in accordance with an illustrative embodiment; and

FIG. 13 is a flowchart of a process of performing a write operationusing dual paths for data replication, in accordance with anillustrative embodiment.

DETAILED DESCRIPTION OF THE INVENTION

As will be appreciated by one skilled in the art, the present inventionmay be embodied as a system, method or computer program product.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present invention may take the form of a computer program productembodied in any tangible medium of expression having computer-usableprogram code embodied in the medium.

Any combination of one or more computer-usable or computer-readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non-exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CDROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.Note that the computer-usable or computer-readable medium could even bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer-usable medium may include a propagated data signal with thecomputer-usable program code embodied therewith, either in baseband oras part of a carrier wave. The computer-usable program code may betransmitted using any appropriate medium, including but not limited towireless, wire line, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJava, Smalltalk, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

The present invention is described below with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions.

These computer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer, orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer program instructions may also bestored in a computer-readable medium that can direct a computer or otherprogrammable data processing apparatus, to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer, or other programmableapparatus to produce a computer-implemented process such that theinstructions which execute on the computer or other programmableapparatus, provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

Illustrative embodiments provide mechanisms for configuration of amulti-root input/output virtualization (MR-IOV) adapter and input/outputfabric to allow for multiple paths from an input/output virtualizationfunction to separate systems. While illustrative embodiments will bedescribed with regard to peripheral component interconnect express(PCIe) adapters or endpoints, the present invention is not limited tosuch. Rather, the mechanisms of the illustrative embodiments may beimplemented in any input/output fabric that supports input/outputvirtualization within the input/output adapters.

Moreover, while illustrative embodiments will be described in terms ofan implementation in which a hypervisor is utilized, the presentinvention is not limited to such. To the contrary, other types ofvirtualization platforms other than a hypervisor, whether implemented insoftware, hardware, or any combination of software and hardware,currently known or later developed, may be used without departing fromthe spirit and scope of the present invention.

With reference now to the figures, and in particular with reference toFIG. 3, a block diagram of a distributed computing system utilizing aperipheral component interconnect multi-root input output fabric isillustrated in accordance with an illustrative embodiment of the presentinvention. FIG. 3 enhances the configurations of FIG. 1 and FIG. 2 withthe addition of peripheral component interconnect fabric to connectsystem nodes with shared input/output adapters. As shown in FIG. 3,distributed computer system 300 comprises plurality of root nodes360-363 coupled to peripheral component interconnect multi-root inputoutput fabric 344 which in turn is coupled to multi-root input outputfabric configuration manager 364 and peripheral component interconnectoutput adapters or endpoints 345-347. Each root node 360-363 comprisesone or more corresponding root complexes 308, 318, 328, 338, and 339,attached to peripheral component interconnect multi-root input/outputfabric 344 through input/output links 310, 320, 330, 342, and 343,respectively, and further attached to memory controllers 304, 314, 324,and 334 of root nodes (RNs) 360-363. Input/output fabric 344 is attachedto input output adapters 345, 346, and 347 through links 351, 352, and353. Input output adapters 345, 346, and 347 may be non-input/outputvirtualization enabled adapters such as in peripheral componentinterconnect express input/output adapter 345, single-root (SR) inputoutput virtualization adapters such as in peripheral componentinterconnect express input/output adapter 346, or multiple-root inputoutput virtualization adapters such as in peripheral componentinterconnect express input/output adapter 347.

As shown, root complexes 308, 318, 328, 338, and 339 are part of rootnodes 360, 361, 362, and 363. More than one root complex per root nodemay be present, such as is shown in root node 363. A root complex is theroot of an input/output hierarchy that connects the centralprocessor/memory to the input/output adapters. The root complex includesa host bridge, zero or more root complex integrated endpoints, zero ormore root complex event collectors, and one or more root ports. Eachroot port supports a separate input/output hierarchy. The input/outputhierarchies may be comprised of a root complex, for example, rootcomplex 308, zero or more interconnect switches and/or bridges (whichcomprise a switch or peripheral component interconnect express fabric,such as peripheral component interconnect multi-root input output fabric344), and one or more endpoints, such as peripheral componentinterconnect express input/output adapters or endpoints 345-347.

In addition to the root complexes, each root node consists of one ormore central processing units 301, 302, 311, 312, 321, 322, 331, and332, memory 303, 313, 323, and 333, memory controller 304, 314, 324, and334. Memory controller 304, 314, 324, and 334 connects centralprocessing units 301, 302, 311, 312, 321, 322, 331, and 332, with memory303, 313, 323, and 333, by way of buses 305, 306, 307, 315, 316, 317,325, 326, 327, 335, 336 and 337 and input/output root complexes 308,318, 328, 338, and 339 by buses 309, 319, 329, 340 and 341. Memorycontrollers typically perform functions such as handling coherencytraffic for the memory. Root nodes 360 and 361 may be connected togetherat connection 359 through their memory controllers 304 and 314 to formone coherency domain. Thus, root nodes 360-361 may act as a singlesymmetric multi-processing (SMP) system, or may be independent nodeswith separate coherency domains as in root nodes 362 and 363.

The multi-root input output fabric configuration manager 364 may beisolated from the other operations of the root nodes, and is thereforeshown as attached separately to input/output fabric 344. However, thisadds expense to the system, and therefore the embodiments as disclosedherein may include this functionality as part of one or more of the rootnodes 360, 361, 362, and 363. Configuration manager 364 configures theshared resources of the multi-root input output fabric 344 and assignsresources to root nodes 360, 361, 362, and 363.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIG. 3 may vary. For example, other peripheral devices, suchas optical disk drives and the like, also may be used in addition to orin place of the hardware depicted. The depicted example is not meant toimply architectural limitations with respect to the present invention.

Using the example of distributed computing system 300 of FIG. 3,illustrative embodiments provide a capability for a single function ofan input/output virtualization device to gain access to multiplesystems. The capability enables configuring, by configuration manager364, an input/output subsystem with redundant paths, allowing the singlefunction of an input/output virtualization device to access multiplesystems through multiple communications paths between the multiplesystems.

Illustrative embodiments address the situation where input/output (I/O)fabric 344 is shared by more than one system such as systems of rootnodes 360, 361, 362 and 363 or logical partition (LPAR), where eachsystem or logical partition can potentially share with the other logicalpartition an input/output adapter (IOA) such as peripheral componentinterconnect express input/output adapters or endpoints 345-347, andwhere multiple systems can share an input/output adapter by use of anmulti-root input/output virtualization fabric. The illustrativeembodiments define a mechanism for a single function of an input/outputvirtualization adapter, such as peripheral component interconnectexpress input/output adapter 347, to be authorized to access multiplesystems or logical partitions of the root nodes while also preventingaccess to systems to which it should not be allowed to access. A singleinput/output virtualization function is thus allowed to access multiplevirtual hierarchies (VHs), or paths, and virtual functions of multi-rootinput/output fabric 344 for the purpose of establishing redundant pathsbetween endpoints 345-347 and memory 303, 313, 323 and 333 of themultiple root nodes.

With reference now to FIG. 4, a block diagram of the virtualization ofsystem resources using multiple logical partitions in which illustrativeembodiments of the present invention may be implemented, is presented.The hardware in logical partitioned platform 400 may be implemented, forexample, within root nodes 360, 361, 362, 363 in FIG. 3, and may furtherinclude portions of multi-root input output fabric 344 and input/outputadapters 345-347 which are assigned to the root node.

Logical partitioned platform 400 includes partitioned hardware 430,operating systems 402, 404, 406, and 408, and partition management ofplatform firmware 410. Operating systems 402, 404, 406, and 408 may bemultiple copies of a single operating system or multiple heterogeneousoperating systems simultaneously run on logical partitioned platform400.

Operating systems 402, 404, 406, and 408 are located in partitions 403,405, 407, and 409. Hypervisor software, or firmware, is an example ofsoftware that may be used to implement partition management of platformfirmware 410. Firmware is “software” stored in a memory chip that holdsits content without electrical power, such as, for example, in aread-only memory (ROM), programmable read-only memory (PROM), erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), and nonvolatile random accessmemory (NVRAM).

Additionally, partitions 403, 405, 407, and 409 also include partitionfirmware 411, 413, 415, and 417. Partition firmware 411, 413, 415, and417 may be implemented using initial boot strap code, for exampleInstitute of Electrical and Electronics Engineers, Inc (IEEE) 1275Standard Open Firmware, and runtime abstraction software (RTAS). Whenpartitions 403, 405, 407, and 409 are instantiated, a copy of boot strapcode is loaded onto partitions 403, 405, 407, and 409 by platformfirmware 410. Thereafter, control is transferred to the boot strap codewith the boot strap code then loading the open firmware and runtimeabstraction software. The processors associated or assigned topartitions 403, 405, 407, and 409 are then dispatched to the partition'smemory to execute partition firmware 411, 413, 415, and 417.

Partitioned hardware 430 includes plurality of processors 432, 434, 436,and 438, a plurality of system memory units 440, 442, 444, and 446,plurality of input output adapters 448, 450, 452, 454, 456, 458, 460,and 462, storage unit 470, and non-volatile random access memory storage498. Each of processors 432, 434, 436, and 438, memory units 440, 442,444, and 446, non-volatile random access memory storage 498, and inputoutput adapters 448, 450, 452, 454, 456, 458, 460, and 462, or partsthereof, may be assigned to one of multiple partitions within logicalpartitioned platform 400, each of which corresponds to one of operatingsystems 402, 404, 406, and 408.

Platform firmware 410 performs a number of functions and services forpartitions 403, 405, 407, and 409 to create and enforce the partitioningof logical partitioned platform 400. Platform firmware 410 may includepartition management firmware which may include a firmware implementedvirtual machine identical to the underlying hardware. Thus, partitionmanagement firmware in platform firmware 410 allows the simultaneousexecution of independent operating system images 402, 404, 406, and 408by virtualizing the hardware resources of logical partitioned platform400.

Service processor 490 may be used to provide various services, such asprocessing of platform errors in partitions 403, 405, 407, and 409.These services also may act as a service agent to report errors back toa vendor. Operations of partitions 403, 405, 407, and 409 may becontrolled through a hardware management console, such as hardwaremanagement console 480. Hardware management console 480 is a separatedistributed computing system from which a system administrator mayperform various functions, including reallocation of resources todifferent partitions. Operations which may be controlled include thingslike the configuration of the partition relative to the components whichare assigned to the partition, whether the partition is running or not.

In a logical partitioning (LPAR) environment, it is not permissible forresources or programs in one partition to affect operations in anotherpartition. Furthermore, to be useful, the assignment of resources needsto be fine-grained. For example, it is often not acceptable to assignall input output adapters under a particular peripheral componentinterconnect host bridge (PHB) to the same partition, as that willrestrict configurability of the system, including the ability todynamically move resources between partitions.

Accordingly, some functionality is needed in the bridges that connectinput/output adapters to the input/output bus so as to be able to assignresources, such as individual input/output adapters or parts ofinput/output adapters to separate partitions; and, at the same time,prevent the assigned resources from affecting other partitions such asby obtaining access to resources of the other partitions.

With reference to FIG. 5A, a block diagram of a peripheral componentinterconnect express multi-root input output virtualization enabledendpoint is presented. As shown in FIG. 5A, peripheral componentinterconnect express multi-root input output virtualization endpoint500, such as multi-root peripheral component interconnect expressinput/output adapter 347 of FIG. 3, includes peripheral componentinterconnect express port 501 through which communications withperipheral component interconnect express switches, and the like, of aperipheral component interconnect express fabric may be performed.Internal routing 502 provides communication pathways to configurationmanagement function 503 and configuration management function 509 and aplurality of virtual functions (VFs) 504-506. Configuration managementfunction 503 may be a physical function (PF) as opposed to virtualfunctions 504-506 and configuration management function 509 may be abase function (BF) 509. A physical “function,” as the term is used inthe peripheral component interconnect specifications, is a set of logicthat is represented by a single configuration space. In other words, aphysical “function” is circuit logic that is configurable based on datastored in the function's associated configuration space in a memory,such as may be provided in non-separable resources 507, for example. Asimilar statement can be made for base “function” 509.

Configuration management function 503 may be used to configure virtualfunctions 504-506. The virtual functions are functions, within aninput/output virtualization enabled endpoint, that share one or morephysical endpoint resources; for example, a link, and which may beprovided in sharable resource pool 508 of peripheral componentinterconnect express input/output virtualization endpoint 500, forexample, with another function. The virtual functions can, withoutrun-time intervention by a hypervisor, directly be a sink forinput/output and memory operations from a system image, and be a sourceof direct memory access (DMA), completion, and interrupt operations to asystem image.

Multi-root input output virtualization endpoint 500 can also be sharedbetween multiple root nodes, for example root nodes 360-363 in FIG. 3.Configuration management function, 509, as a base function, may be usedto configure characteristics of the physical functions, for example,which root node has access to each physical function.

Peripheral component interconnect express endpoints may have manydifferent types of configurations with regard to the “functions”supported by the peripheral component interconnect express endpoints.For example, endpoints may support a single physical function, multipleindependent physical functions, or even multiple dependent physicalfunctions. In endpoints that support native input/output virtualization,each physical function supported by the endpoints may be associated withone or more virtual functions, which themselves may be dependent uponvirtual functions associated with other physical functions. The unit ofthe input output virtualization endpoint which is assigned to a rootnode is the physical function, and multi-root input outputvirtualization enabled endpoints will contain multiple physicalfunctions.

In one embodiment, virtual function (VF) to virtual hierarchy (VH)authorization tables 510 allow configuration manager 364 of FIG. 3 togive each function access to multiple virtual hierarchies. This aspectwill be described later. Virtual function work queues 511, also to bedescribed further, are setup by the device driver software for thevirtual function and specify the operations to be performed by thevirtual function. The virtual function work queue entries in the tablewill also include the virtual hierarchy number or numbers to use for theparticular operation being requested.

With reference to FIG. 5B, a block diagram of a peripheral componentinterconnect express multi-root enabled peripheral componentinterconnect express switch, is presented. Peripheral componentinterconnect express switch 520 might be used, for example in peripheralcomponent interconnect multi-root input/output fabric 344 in FIG. 3, asdefined by the peripheral component interconnect multi-root input/outputvirtualization specification. Switch 520 logically consists of multiplevirtual planes, one per port that is connected to a root node. Forexample, root node 521 connects, by peripheral component interconnectexpress link 524, to the logical peripheral component interconnect toperipheral component interconnect (P2P) bridge 527 which is logicallyconnected internally to the switch to peripheral component interconnectto peripheral component interconnects 536-538. Similarly, root node 522connects, by peripheral component interconnect express link 525, to thelogical peripheral component interconnect to peripheral componentinterconnect bridge 528 which is logically connected internally to theswitch to peripheral component interconnect to peripheral componentinterconnect 530-532, and root node 523 connects, by peripheralcomponent interconnect express link 526, to the logical peripheralcomponent interconnect to peripheral component interconnect bridge 529which is logically connected internally to the switch to peripheralcomponent interconnect to peripheral component interconnect 533-535.

Peripheral component interconnect to peripheral component interconnectbridges 530, 533, and 536 then share peripheral component interconnectexpress multi-root link 539 so that they can share the resources ofmulti-root peripheral component interconnect express device 542. In asimilar manner, peripheral component interconnect to peripheralcomponent interconnect bridges 531, 534, and 537 then share peripheralcomponent interconnect express multi-root link 540 so that they canshare the resources of peripheral component interconnect expressmulti-root device 543, and peripheral component interconnect toperipheral component interconnect bridges 532, 535, and 538 then shareperipheral component interconnect express multi-root link 541 so thatthey can share the resources of multi-root peripheral componentinterconnect express device 544.

The control point for setting up switch 520 is base function (BF) 545.This input/output virtualization configuration mechanism, for example,base function 545, allows a multi-root peripheral component interconnectmanager (MR-PCIM) program to determine the logical structure withinswitch 520. For example, FIG. 5B shows a fairly symmetric configuration,with each root node 521-523 having access to part of each peripheralcomponent interconnect express multi-root device 542-544. In normalsystems, the system administrator may want to setup the input/output ina less symmetric way, in order to meet the needs of the users using thesystem.

Base functions 545 and 509 are accessed by a multi-root peripheralcomponent interconnect manager program. Where this program resides isnot specified by the peripheral component interconnect special interestgroup input/output virtualization specifications. The program couldreside, for example, in a node that is dedicated solely to a multi-rootperipheral component interconnect manager and is attached to one of theroot port nodes, as is shown by one of root nodes 521-523, or may beprovided via a vendor-unique port with a separate processor attached,for example, a service processor as in 490 in FIG. 4. Regardless ofwhere the multi-root peripheral component interconnect manager isexecuted, the main requirement is that this program be robust and cannotbe affected by the operations, or failure thereof, of other applicationsin the system.

Illustrative embodiments provide a mechanism for configuration of aninput/output virtualization adapter, such as input/output virtualizationenabled peripheral component interconnect express endpoint 500 shown inFIG. 5A, to access more than one system. The mechanisms of theillustrative embodiments address the situation where an input/outputfabric, which may comprise one or more peripheral component interconnectexpress switches such as peripheral component interconnect expressswitch 520 in FIG. 5B, is shared by more than one system, for exampleroot nodes 362 and 363 of FIG. 3.

With reference now to FIG. 6A, a block diagram of a virtual function(VF) work queue entry, in accordance with an illustrative embodiment, ispresented. The example provided is representative of virtual functionwork queue entry 511 in FIG. 5. Fields 605 and 607 of virtual functionwork queue entry 601 contain the peripheral component interconnectexpress fabric virtual hierarchy numbers. Fields 605 and 607 indicate tothe virtual function which system to send to or from which system toreceive the direct memory access data for the operation. The fieldsallow the device driver software to send the same data to multiplesystems. For example, on a network adapter a packet received from avirtual function network connection may be written to the system memoryof multiple systems. For example, to system memory 323 and 333 insystems 362 and 363 in FIG. 3, in order to maintain a copy of the dataif a system fails before data is committed to a non-volatile storagemedia such as a disk. By having redundancy a system can signal that thedata has been committed to disk, for example, before the data isactually written to the disk, because the data is safe in a redundantsystem. Signaling a completion of the write operation before the data iswritten physically to disk provides a faster response, less latency, tothe requester of the disk write operation.

Other fields of virtual function work queue entry 601 include operationtype 602, transfer length 603, and operation addresses 604, 606.Operation type 602 indicates what operation to perform to the virtualfunction. For example, for a network adapter, the operation may be toset up receive buffers. In this case, the receive buffer may be setup inmore than one system using more than one operation address andperipheral component interconnect express fabric virtual hierarchynumber pair of fields, one pair for each system. There is one pair ofthese fields, for example 604 and 605, 606 and 607, for each system forwhich to send the received data. Transfer length 603, in this case,would be set to the buffer length.

Those skilled in the art will recognize that the types of operations andthe field types may vary by the functionality to be provided by theadapter. The peripheral component interconnect express fabric virtualhierarchy number is provided for each address, in order to direct thedata to the correct system.

With reference to FIG. 6B, a block diagram of tables for validating theauthority of a virtual function to access any given virtual hierarchy ina multi-root device, in accordance with an illustrative embodiment, ispresented. Virtual function to virtual hierarchy authorization tables610 is an example of virtual function to virtual hierarchy authorizationtables 510 of FIG. 5A. In a multi-root device the adapter provides theequivalent of a firewall between functions that can be accessed bydifferent systems. Peripheral component interconnect express fabricvirtual hierarchy number fields 605, and 607, in FIG. 6A, provide amechanism for tunneling through a firewall to use the virtual hierarchynumber that would normally be assigned to a different functioncontrolled by a different system. Since peripheral componentinterconnect express fabric virtual hierarchy number fields field 605and 607 in FIG. 6A are setup by device driver software in one system, itis important that the virtual hierarchy number used is validated, sothat a system can set up an associated virtual function to only tunnelthrough allowed firewalls on the adapter. The required functionality isprovided through virtual function to virtual hierarchy authorizationtables 610. There is a virtual function number to virtual hierarchyauthorization table 611 and 615, for each virtual function in theadapter. In the example, the table may include multiple entries 612-614,616-618, one entry for each virtual hierarchy that the virtual function,to which the table applies, is allowed to access. Prior to allowing avirtual function to process a virtual function work queue entry 601,peripheral component interconnect express fabric virtual hierarchynumber fields 605, 607 are checked against the appropriate virtualfunction to virtual hierarchy authorization table to make sure that thevirtual function has authority to access the virtual hierarchy number.If not authorized, the processing of virtual function work queue entry601 is not allowed, and an error is signaled to the device driversoftware. Virtual function to virtual hierarchy authorization tables 610are setup by trusted software. For example the trusted software may be amulti-root input/output fabric configuration manager or multi-rootperipheral component interconnect manager 364 in FIG. 3. The tablecannot be changed by the device driver software in the systems, thusmaking the control of the tunneling process secure. Further explanationof the use of these tables will be described later.

With reference to FIG. 6C, a block diagram of a table for specifying analternate route virtual hierarchy for redundant path implementations ofa multi-root device, in accordance with an illustrative embodiment, ispresented. Authorized virtual hierarchy number for virtual functiontables 620 a correspondence between primary and secondary path entriesmaintained in internal routing 502 of FIG. 5A. If one of the pathsspecified by the virtual hierarchy number in virtual function to virtualhierarchy authorization tables 610 becomes unavailable, a redundant androbust configuration provides a capability to use an alternate path tothe desired system for the operation. Expanded authorized virtualhierarchy number for virtual function tables 620 can be used instead ofvirtual function to virtual hierarchy authorization tables 610, in thiscase. The difference in authorized virtual hierarchy number for virtualfunction tables 620 is that for each entry 621-625, there is analternate entry 622-626 specifying an alternate virtual hierarchy numberto use in place of the virtual hierarchy number that is non-operational.For example, if entry 621 specifies virtual hierarchy number “1” andentry 622 specifies virtual hierarchy number “3,” when virtual functionwork queue entry 601 specifies virtual hierarchy number “1” and virtualhierarchy number “1” is detected as non-operational, then virtualhierarchy number “3” can be used to access the same system memory in thesame system as would have been available with virtual hierarchy number“1”. Thus, not only is there a way to avoid entire system failures, forexample, by sending the same data to multiple systems for the sameoperation, but there is also a way to avoid input/output fabric failuresthrough redundancy. The virtual hierarchy number may be used as a pathidentifier.

With reference to FIG. 6D, a block diagram of a table for specifying anauthorized address to virtual function relationship, in accordance withan illustrative embodiment, is presented. Virtual function to addressauthorization table 628 contains a table for each virtual functionrequiring authorization. For each function, a set of permitted addressesis provided, with entries in the table 630, 640 representing a range ofaddresses that the associated virtual function is allowed to access. Inthe example, the table for the first virtual function 630 has a set ofentries associated. Addresses that the first virtual function 630 ispermitted to use are listed as authorized addresses 632-638. In asimilar manner a last virtual function “VFn” has a set of entriesdepicted by table entry 640. The function of virtual function to addressauthorization tables 628 is similar to that of virtual function tovirtual hierarchy authorization tables 610 of FIG. 6B in permittingaccess by a virtual function to resources, for example address ranges indifferent logical partitions of the same root node. Virtual function toaddress authorization table 628 provides a capability similar to virtualfunction to virtual hierarchy authorization table 610, with addressesrather than virtual hierarchies being used.

With reference to FIG. 6E, a block diagram of a virtual function (VF)work queue entry using an address of FIG. 6D, in accordance with anillustrative embodiment, is presented. The example provided isrepresentative of virtual function work queue entry 601 in FIG. 6A, as afurther example of virtual function work queues 511 of FIG. 5. In thisexample, virtual function work queue entry 642 contains a number offields including operation type 644, and transfer length 646 as before.A difference from the prior virtual function work queue entry of FIG. 6Ais that there are no virtual hierarchy numbers. In place of the virtualhierarchy numbers are found operation address 648 through operationaddress 650. The operation address specifies a location associated withthe data, for example, addresses within different logical partitions ofthe same root node.

With reference to FIG. 7, a block diagram of a configuration ofredundant systems using multi-root devices and multi-root switches, inaccordance with an illustrative embodiment, is presented. Using theexample of distributed computing system 300 of FIG. 3, a configurationof systems 700 using multi-root devices and multi-root switchesconnected using computer electronic complex (CEC) to computer electroniccomplex communication devices such as multi-root device 1 727 andmulti-root device 2 728, is defined.

Two computer systems are shown, comprising computer electronic complex 1701 and computer electronic complex 2 702, but those skilled in the artwill recognize that more than a two-way redundant system could beconstructed. The computer electronic complexes correspond to the rootnodes in FIG. 3 with the peripheral component interconnect host bridges(PHB) corresponding to the root complexes of FIG. 3.

The two computer electronic complexes may also be partitioned as in FIG.4 to form sets of logical partitions. The two computer electroniccomplexes consist of system memory 703, 704, and three peripheralcomponent interconnect host bridges each, 705-707 and 708-710.Multi-root peripheral component interconnect manager 711 corresponds tothe configuration manager 364 in FIG. 3. This being a highly redundantsystem, there also is a backup multi-root peripheral componentinterconnect manager 712 which can take over for the primary multi-rootperipheral component interconnect manager 711 in case of the failure ofthe primary multi-root peripheral component interconnect manager 711,failure of computer electronic complex 1, failure of peripheralcomponent interconnect host bridge 1 (PHB1) 705, or any other failurethat prevents multi-root peripheral component interconnect manager 711from controlling the multi-root input/output fabric operations. Themulti-root peripheral component interconnect manager fail-over processis beyond the scope of this invention.

The multi-root peripheral component interconnect managers 711 and 712are connected to virtual hierarchy 0 of the multi-root fabric, which isdefined by the peripheral component interconnect express multi-rootinput/output virtualization specification as being the managementvirtual hierarchy, though peripheral component interconnect host bridge1 (PHB1) 705 and peripheral component interconnect express link 713 tomulti-root switch 1 719 and through peripheral component interconnecthost bridge 6 (PHB6) 710 and peripheral component interconnect expresslink 716 to multi-root switch 2 720. The other peripheral componentinterconnect host bridges form a primary virtual hierarchy connectionand secondary virtual hierarchy connection to the multi-root fabric.Specifically, computer electronic complex 1 primary virtual hierarchy isvirtual hierarchy 1 and computer electronic complex 1 connects tovirtual hierarchy 1 through peripheral component interconnect hostbridge 2 (PHB2) 706 through peripheral component interconnect expresslink 714 to multi-root switch 1 719. Computer electronic complex 1secondary virtual hierarchy connection is virtual hierarchy 3 connectingto virtual hierarchy 3 through peripheral component interconnect hostbridge 3 (PHB3) 707 through peripheral component interconnect expresslink 718 to multi-root switch 2 720. Similarly, computer electroniccomplex 2 primary virtual hierarchy is virtual hierarchy 4 connecting tovirtual hierarchy 4 through peripheral component interconnect hostbridge 5 (PHB5) 709 through peripheral component interconnect expresslink 715 to multi-root switch 2 720. Computer electronic complex 2secondary virtual hierarchy connection is virtual hierarchy 2 connectingto virtual hierarchy 2 through peripheral component interconnect hostbridge 4 (PHB4) 708 through peripheral component interconnect expresslink 717 to multi-root switch 1 719.

The “secondary” link is not necessarily just for backup purposes, but isalso used for communications to devices depending on the switch underwhich the devices are located. Typically the shortest path from deviceto computer electronic complex is used, which is the path through thefewest number of switches, to reduce the operational latency. A paththrough multiple switches would then typically be reserved for backuppurposes. Peripheral component interconnect express links 721, 722provide the cross-switch connections to provide alternate paths.

Below each multi-root switch is shown two multi-root devices. Multi-rootdevice 1 727, is shown as a network device that connects to the networkby connection 738 multi-root switch 1 727 via peripheral componentinterconnect express link 723. Similarly, multi-root device 2 728 isshown as a network device that connects to the network via connection739 and to multi-root switch 2 via peripheral component interconnectexpress link 726.

For redundancy reasons, the two network adapter connections 738, and 739would most likely connect to an external network switch and both deviceswould have access to the same network. That way, if one network adapterfailed, both central electronic complexes would still have access to thenetwork via the remaining adapter. In addition to the network adapters727, 728, are two disk adapters, as multi-root device 3 729 andmulti-root device 4 730, with both of these devices given access to thesame set of disk drives 731. Multi-root device 3 729 is connected viaperipheral component interconnect express link 724 to multi-root switch1 719 and multi-root device 4 730 is connected via peripheral componentinterconnect express link 725 to multi-root switch 2 720.

In this example, multi-root device 1 727 has access to four virtualhierarchies, namely virtual hierarchy 1 732, virtual hierarchy 2 733,virtual hierarchy 3 734, and virtual hierarchy 4 735. Each of thesevirtual hierarchies would normally be associated with a separateperipheral component interconnect express function. For example, virtualfunctions, in which each of the functions would be separated byfirewalls 737 such that one virtual function could not get access to avirtual hierarchy of another virtual function. Firewall tunnel 736 maybe created between virtual hierarchy 1 732 and virtual hierarchy 2 733(for example, between virtual function 1 and virtual function 2 ofmulti-root device 727), allowing multi-root device 1 727 to directmemory access data to memory 703, and memory 704 in both computerelectronic complexes which are connected to different sets of virtualhierarchies.

Multi-root device 1 727 is logically similar to peripheral componentinterconnect express multi-root input/output virtualization end point500 shown in FIG. 5A. As such, it contains virtual function to virtualhierarchy authorization tables 510 in FIGS. 5A and 610 in FIG. 6B andvirtual function work queues 511 in FIG. 5A with virtual function workqueue entries 601 in FIG. 6A. Trusted software as inmulti-root—peripheral component interconnect manager 711 has setup thevirtual function to virtual hierarchy authorization tables to allow avirtual function to get access to both virtual hierarchy 1 732 andvirtual hierarchy 2 733, essentially forming a tunnel through firewalltunnel 736.

Other embodiments of a tunnel through the firewall may be used. Forexample, a capability for one virtual function to create a communicationpath to another virtual function by some means and pass the informationto the other virtual function, along with the operation to perform onthe data may be provided. The other means would also require a securemethod of setting up such means, like the mechanism described, so thatthe tunnel through the firewall could be controlled by a trusted pieceof code.

The following describes an operation of receiving data from network link738 which is destined to be written to disk. A device driver in computerelectronic complex 1 701 which is responsible for handling the virtualfunction creates receive buffers in system memory 703. In addition,computer electronic complex 1 701 has communicated with a correspondingdriver in computer electronic complex 2 702, for example by using anetwork connection between the two computer electronic complexes. Thecorresponding computer electronic complex 2 702 driver has allocatedcorresponding receive buffers in system memory 704 and then hascommunicated the address of the receive buffers to the driver incomputer electronic complex 1 701. The driver in computer electroniccomplex 1 701 then sets up a virtual function work queue entry in thevirtual function of multi-root device 1 727 that points to the computerelectronic complex 1 701 receive buffer via virtual hierarchy 1 732 andthe computer electronic complex 2 702 receive buffer via virtualhierarchy 2 733. Upon receiving a communication packet by the virtualfunction, the virtual function uses the information given in the virtualfunction work queue entry to identify where the buffers are located, andthen verifies the authority of the virtual function to tunnel throughthe firewall by use of the virtual function to virtual hierarchyauthorization table 610 in FIG. 6B that corresponds to the virtualfunction. If the authorization passes, multi-root device 1 727 thendirect memory accesses the data via virtual hierarchy 1 732 into systemmemory of central electronic complex 1 701 and then direct memory accessthe data via virtual hierarchy 2 733 into system memory 704 of computerelectronic complex 2 702. On successful completion of these directmemory accesses, the device driver gets signaled by an interrupt frommulti-root device 1 727 and detects the operation completed successfullyto both computer electronic complexes. At this point the data is safefrom a failure in one of the computer electronic complexes, and theoperation can be considered to have successfully completed. At thispoint the originator of the disk write operation is told it is complete,even though it is still not on disk, because the data is safe from asystem crash, for example. The device driver in computer electroniccomplex 1 701 then queues up a disk write operation through multi-rootdevice 3 729.

With further reference to FIG. 7, multiple paths through the multi-rootfabric consisting of the two multi-root switches are presented. Forexample, if there had been a failure of link virtual hierarchy 717, thenthe multi-root device would not be able to perform a write to systemmemory 704 as described above. If the multi-root device implements theredundant table shown in FIG. 6C, then when the path from multi-rootdevice 1 727 to computer electronic complex 2 702 through that link isnot operational, the table shown in FIG. 6C can be used to determinethere is an alternate path by virtual hierarchy 4 735 instead of virtualhierarchy 2 733, and the data would flow through link 723 throughmulti-root switch 1 719 through peripheral component interconnectexpress links 721, 722 through multi-root switch 2 720, throughperipheral component interconnect express link 715, through peripheralcomponent interconnect host bridge 5 (PHB5) 709 to system memory 704.

With reference to FIG. 8, a block diagram of a configuration ofredundant logical partitions using only a single root device, inaccordance with an illustrative embodiment. As a further example oflogical partitioned platform 400 of FIG. 4, configuration 800 ispresented in which there is no concept of multiple virtual hierarchies.Instead of having separate virtual hierarchies, there is a concept ofhaving direct memory access address ranges assigned to the virtualfunctions. Single system 801 consists of multiple logical partitions802-803, each with one or more central processing units 804-807, andeach central processing unit with memory 808-809. The logical partitionsshare one or more peripheral component interconnect host bridges (PHBs)810-811, and single root devices 814-815 are connected to the peripheralcomponent interconnect host bridges through peripheral componentinterconnect express links 812-813. The single root devices are shown asnetwork adapters, and the devices are connected to the network throughlinks 816-817. As in FIG. 7, the two network links would be connected toa network switch (not shown), so that both links could get access to thesame network if the other link failed. Also, as in the FIG. 7, virtualfunctions 818-821 are separated by firewalls 823, and firewall tunnel822 is created to permit a virtual function to access the logicalpartition memory of another virtual function. The access differs fromthe standard peripheral component interconnect express input/outputvirtualization specification which requires each virtual function toaccess the memory of one and only one logical partition.

The data structures that allow the single-root tunneling are similar towhat is needed for the multi-root case, which are shown in FIG. 6B.Instead of the tables containing the virtual hierarchy, each authorizedvirtual hierarchy number is replaced by an authorized peripheralcomponent interconnect express direct memory access address range. Thesingle-root peripheral component interconnect manager, (not shown),similar to the multi-root peripheral component interconnect manager inthe multi-root case, allocates the peripheral component interconnectexpress address ranges and sets up the virtual function to address rangeauthorization tables 628 in FIG. 6D. The software in the logicalpartitions is not given access to the table, so that one logicalpartition cannot get access to the memory of another logical partition,unless explicitly setup, as it was for the virtual hierarchies in themulti-root case. As in the FIG. 7 multi-root case, the two logicalpartitions communicate in the same manner as the software did in thecomputer electronic complexes of the multi-root case, to setup receivebuffers. Virtual function work queue entry 642 in FIG. 6E does includethe virtual hierarchy number in this case; however the operationaddress, such as 648 in FIG. 6E may be used as a path identifier.

Two logical partitions are shown in FIG. 8, but those skilled in the artwill recognize that more than a two-way redundant set of logicalpartitions could be constructed. As shown, the virtual function may beone of a multi-root peripheral component interconnect device virtualfunction and a single root peripheral component interconnect devicevirtual function.

With reference to FIG. 9, a flowchart of a process of multi-root fabricconfiguration of a multi-root multi-system configuration in accordancewith an illustrative embodiment is presented. Configuration process 900is an example of a configuration process of configuration manager 364 ofFIG. 3 providing a configuration as shown in FIG. 7. Configurationprocess 900 starts (step 902) and the multi-root peripheral componentinterconnect manager configures the multi-root fabric (step 904).Configuring the multi-root fabric creates correct routes from devices toroot complexes, including any desired alternate routes for redundancy.The multi-root peripheral component interconnect manager makes availableto the root complexes the virtual hierarchy numbers to peripheralcomponent interconnect host bridge (PHB) correlation (step 906). Themulti-root peripheral component interconnect manager invokes a devicedriver for device physical functions to set up virtual function tovirtual hierarchy numbers authorization tables, including any alternatecorrelations (step 908) with configuration process 900 terminatingthereafter.

With reference to FIG. 10, a flowchart of a process allowing a system todetermine the virtual hierarchy numbers required for communicating topartner systems, in accordance with an illustrative embodiment ispresented. Process 1000 is as example of a process using theconfiguration of FIG. 7 by root node 360 and root node 361 or theconfiguration of FIG. 8 and logical partition 403 and logical partition405 of FIG. 4.

Process 1000 starts (step 1002) and the computer electronic complexescommunicate with one another or when logical partitions are used,logical partitions communicate with one another to discover respectivepartners and the virtual hierarchy numbers associated with a partner(step 1004). Each of the computer electronic complexes or logicalpartitions discover the devices associated with the respective complexor partition, load the device drivers for their respective discovereddevices, and read the virtual function to virtual hierarchy numberauthorization table for their respective virtual functions (step 1006).The device drivers now have the virtual hierarchy numbers needed tosetup the appropriate virtual function work queue entries 601 of FIG.6A. Process 1000 terminates thereafter (step 1008).

With reference to FIG. 11, a flowchart of a process to setup of avirtual function work queue entry in accordance with one illustrativeembodiment is presented. Process 1100 is an example of process toestablish a virtual function work queue entry 511 of FIG. 5 by centralelectronic complex, such as CEC 1 701 of FIG. 7 or LPAR 1 802 of FIG. 8.Process 1100 starts (step 1102) and the master computer electroniccomplex or logical partition sets up the virtual function work queueentry in the system virtual function (step 1104). The entry createdspecifies the virtual hierarchy number for all computer electroniccomplexes or logical partitions to which the operation is applicable.The master computer electronic complex or logical partition is where thedevice driver resides for a particular operation. All computerelectronic complexes or logical partitions can have master operationsexecuting simultaneously. That is, one computer electronic complex orlogical partition may take part of the workload and control that part,and another computer electronic complex or logical partition may takeanother part of the workload, in order to spread the workloads betweenthe various computer electronic complexes or logical partitions.

The device performs the requested operation, sending the data to thesystem memory of all appropriate computer electronic complexes orlogical partitions using the virtual hierarchy numbers and addresses inthe virtual function work queue entry for the operation (step 1106).Process 1100 terminates thereafter (step 1108).

With reference to FIG. 12, a flowchart of a process for dynamicallydetermining input/output fabric path operational status and use of analternate path, in accordance with an illustrative embodiment ispresented. Process 1200 is an example of a process of a device, such asMR device 1 727 of FIG. 7 to determine path availability. Process 1200starts (1202) and a device periodically determines the operationalstatus of a virtual hierarchy path to system memory, setting a flag ifthe virtual hierarchy path is not available (step 1204). For example,the device reads a location in system memory via direct memory accessand if the device receives an error on the read, such as an operationtimeout, the device marks the path as not available. Typically thedevice starts an operation, on the primary path if that path isavailable; otherwise the device uses the alternate path (step 1206).Process 1200 terminates thereafter (step 1208).

With reference to FIG. 13, a flowchart of a process of performing awrite operation using dual paths for data replication in accordance withan illustrative embodiment is presented. Process 1300 is an example of adevice driver portion of central electronic complex, such as CEC 1 701of FIG. 7 or LPAR 1 802 of FIG. 8. Process 1300 is an example of a dualpath disk write operation, but more or multiple paths may be used aswell. For example, a write operation may include a set of paths, whereinthe set includes more than two paths. Process 1300 of the storage serverdisk write operation starts at (step 1302) and a device driver sets up avirtual function work queue entry to point to two communication networkreceive buffers; one receive buffer located in each central electroniccomplex (step 1304). The data to be written to disk is received from ahost via the communication network, and is written to the buffers inboth central electronic complexes (step 1306). The write operation toeach receive buffer occurs concurrently or approximately at the sametime. The disk write operation is reported back to the host as havingbeen completed, even though the data has not written to disk yet (step1308). The data is physically written to the disk and the data isdiscarded from the memory of both central electronic complexes after asuccessful completion of the disk write (step 1310). Process 1300terminates thereafter (step 1312).

Illustrative embodiments thus provide a capability for a single functionof an input/output virtualization device to gain access to multiplesystems through multiple paths between the multiple systems. Inparticular, the single function may be permitted access to multiplevirtual hierarchies of the input/output fabric to establish redundantcommunication paths. The establishment of redundant systems enables datato be sent to more than one system of the multiple systems for dataintegrity reasons or to access data in more than one system from thesame function of the same input/output virtualization adapter. In anillustrative embodiment, permission is established though use of virtualfunction to virtual hierarchy authorization correspondence tables orvirtual function to address range authorization tables. Thecorrespondence specifically permits a function to tunnel through afirewall separating virtual functions or virtual hierarchies, to use aresource of another virtual function or virtual hierarchy associatedwith the resource.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

The invention can take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In a preferred embodiment, the invention isimplemented in software, which includes but is not limited to firmware,resident software, microcode, etc.

Furthermore, the invention can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer-readable medium can be any tangibleapparatus that can contain, store, communicate, propagate, or transportthe program for use by or in connection with the instruction executionsystem, apparatus, or device.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Examples of a computer-readable medium include asemiconductor or solid state memory, magnetic tape, a removable computerdiskette, a random access memory (RAM), a read-only memory (ROM), arigid magnetic disk and an optical disk. Current examples of opticaldisks include compact disk read only memory (CD-ROM), compact diskread/write (CD-R/W) and DVD.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modem and Ethernet cards are just a few of thecurrently available types of network adapters.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A computer-implemented method for creating redundant systemconfigurations, the computer-implemented method comprising: creating aset of virtual function path authorization tables, by a trusted entity,wherein entries define access for a virtual function to a set of addressranges in one or more systems and the entries further defining aboundary preventing invalid cross function access, wherein the virtualfunction is performed by a single root or a multi-root peripheralcomponent interconnect device; receiving a request from a requester toprovide requested data from the virtual function; creating a receivebuffer in a selected address range in the set of address ranges;creating a virtual function work queue entry for the virtual functioncontaining an address of the receive buffer in the selected addressrange; determining, in the set of virtual function path tables, whetherthe virtual function is authorized to use the selected address range;responsive to a determination that the virtual function is authorized,writing the requested data into the receive buffer of the selectedaddress range in the one or more systems; and responsive to writing therequested data, issuing a notice of completion to the requester.
 2. Thecomputer-implemented method of claim 1, wherein the set of virtualfunction path authorization tables contains a plurality of entriesdefining a set of paths for a virtual function and address range,wherein a primary entry defines a primary path between the virtualfunction and an address range, and each secondary entry defines analternative path between the virtual function and the address range. 3.The computer-implemented method of claim 1, wherein the set of virtualfunction path authorization tables contains a correspondence betweeneach virtual function and a set of virtual hierarchies.
 4. Thecomputer-implemented method of claim 1, wherein the set of virtualfunction path authorization tables contains a correspondence betweeneach virtual function and a set of address ranges in a set of logicalpartitions.
 5. The computer-implemented method of claim 2, wherein theprimary path is a preferred path, and responsive to the primary pathbeing unavailable, using one of the alternative paths.
 6. Thecomputer-implemented method of claim 1, wherein the presence of an entryin the set of virtual function path authorization tables, created by thetrusted entity, permits access for a function to a system in the set ofsystems by a primary path and a corresponding secondary path.
 7. Thecomputer-implemented method of claim 1, wherein the absence of an entryin the set of virtual function path authorization tables, created by thetrusted entity, prevents access for a function to a system in the set ofsystems by a primary path and a corresponding secondary path.
 8. A dataprocessing system for creating redundant system configurations, the dataprocessing system comprising: a bus; a memory connected to the bus,wherein the memory comprises computer-executable instructions; a centralprocessor unit, wherein the central processor unit executes thecomputer-executable instructions to direct the data processing systemto: create a set of virtual function path authorization tables, by thetrusted entity, wherein entries define access for a virtual function toa set of address ranges in one or more systems and the entry furtherdefining a boundary preventing invalid cross function access, whereinthe virtual function is performed by a single root or a multi-rootperipheral component interconnect device; receive a request from arequestor to provide requested data from the virtual function; create areceive buffer in a selected address range in the set of address ranges;create a virtual function work queue entry for the virtual functioncontaining an address of the receive buffer in the selected addressrange; determine, in the set of virtual function path tables, whetherthe virtual function is authorized to use the selected address range;responsive to a determination that the virtual function is authorized,write the requested data into the receive buffer of the selected addressrange in the one or more systems; and responsive to writing therequested data, issue a notice of completion to the requestor.
 9. Thedata processing system of claim 8, wherein the set of virtual functionpath authorization tables contains a plurality of entries defining a setof paths among each virtual function and address range, wherein aprimary entry defines a primary path between the virtual function and anaddress range, and each secondary entry defines an alternative pathbetween the virtual function and the address range.
 10. The dataprocessing system of claim 8, wherein the set of virtual function pathauthorization tables contains a correspondence between each virtualfunction and a set of virtual hierarchies.
 11. The data processingsystem of claim 8, wherein the set of virtual function pathauthorization tables contains a correspondence between each virtualfunction and a set of address ranges in a set of logical partitions. 12.The data processing system of claim 9, wherein the primary path is apreferred path, and responsive to the primary path being unavailable,using one of the alternative paths.
 13. The data processing system ofclaim 8, wherein the presence of an entry in the set of virtual functionpath authorization tables, created by the trusted entity, permits accessfor the virtual function to a system in the set of systems by a primarypath and a corresponding secondary path.
 14. The data processing systemof claim 8, wherein the absence of an entry in the set of virtualfunction path authorization tables, created by the trusted entity,prevents access for the virtual function to a system in the set ofsystems by a primary path and a corresponding secondary path.
 15. Acomputer program product for creating redundant system configurations,the computer program product comprising: a computer-readable mediumhaving computer-executable instructions stored thereon, thecomputer-executable instructions comprising: computer-executableinstructions for creating a set of virtual function path authorizationtables, by a trusted entity, wherein each entry defines access for avirtual function to a set of address ranges in one or more systems andthe entry further defining a boundary preventing invalid cross functionaccess wherein the virtual function is performed by a single root or amulti-root peripheral component interconnect device computer-executableinstructions for receiving a request from a requester to providerequested data from the virtual function; computer-executableinstructions for creating a receive buffer in a selected address rangein the set of address ranges; computer-executable instructions forcreating a virtual function work queue entry for the virtual functioncontaining an address of the receive buffer in the selected addressrange; computer-executable instructions for determining, in the set ofvirtual function path tables, whether the virtual function is authorizedto use the selected address range; computer-executable instructionsresponsive to a determination that the virtual function is authorized,for writing the requested data into the receive buffer of the selectedaddress range in the one or more systems; and computer-executableinstructions responsive to writing the requested data, for issuing anotice of completion to the requester.
 16. The computer program productof claim 15, wherein the set of virtual function path authorizationtables contains a plurality of entries defining a set of paths amongeach virtual function and address range, wherein a primary entry definesa primary path between the virtual function and an address range, andeach secondary entry defines an alternative path between the virtualfunction and the address range.
 17. The computer program product ofclaim 15, wherein the set of virtual function path authorization tablescontains a correspondence between each virtual function and a set ofvirtual hierarchies.
 18. The computer program product of claim 15,wherein the set of virtual function path authorization tables contains acorrespondence between each virtual function and a set of addressranges.
 19. The computer program product of claim 16, wherein thepresence of an entry in the set of virtual function path authorizationtables, created by the trusted entity, permits access for a function toa system in the set of systems by a primary path and a correspondingsecondary path.
 20. The computer program product of claim 16, whereinthe absence of an entry in the set of virtual function pathauthorization tables, created by the trusted entity, prevents access fora function to a system in the set of systems by a primary path and acorresponding secondary path.